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Abstract: The move to electronic forms of communication has changed the way 
educational research is communicated. In particular it has led to an increased reliance on 
journals. This change has been more precipitous than one might expect. As journals 
become a larger proportion of the body of current scholarship, authors and readers seek 
new methods of determining and communicating journal quality. Methods that have 
worked for researchers in the sciences are not directly transferable to the education 
discipline. This paper examines existing tools and looks for new methods of assessing 
quality in journal publishing. 



Technology has transformed the way information is accessed and transferred. For 
personal communications there are blogs, wikis, and instant messaging. New software- 
like Blackboard or Angel, and podcasting-impacts the way instructors interact with 
classes. It is useful to consider to what extent these new venues of communication change 
professional assessments of the value and quality of educational information. Instructors 
have always endeavored to lead students to high quality information, often by 
recommending specific journals or researchers. For research assignments, where students 
are expected to explore a particular topic and produce a synthesis product, the types of 
materials recommended as high quality sources of information is evolving. Libraries 
note such changes to the extent that they impact collections use and materials selection 
budgets, but teaching faculty may be less aware of the impact. The process of doing a 
literature review is vastly different than it was 20 to 30 years ago. There is an increased 
focus on journal literature, and a concomitant interest in evaluating the quality of journal 
publications. Education is not alone, these changes have happened in most academic 
disciplines to some degree. This paper is an attempt to assess the extent and impact of 
literature format changes in the field of education. First through a look at how the 
presentation of educational information in published form has changed, both by looking 
at the instructions given to students for conducting a literature review and via a quick 
overview of the publishing record. This is followed by a discussion of how these new 
emphases have led to new demands on the ability to assess quality. 
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Changes in teaching about library use 

Books of advice about library research aimed at college students have long been a 
popular mainstay on college campuses. In 1969 Brogan and Buck published Using 
Libraries Effectively. The book was aimed at the “post secondary” audience and 
suggested researchers use the card catalog to find books on their subject. The next step in 
finding materials would be bibliographies, which might be found at the end of 
encyclopedia articles, in one’s textbook, or in other specialized reference books. In 
chapter seven (of twelve) the authors suggest using indexes to articles, but discuss only 
the Readers ’ Guide to Periodical Literature in any depth. The Readers ’ Guide is an 
index to popular magazines. They mention in passing the other specialized indexes of the 
day, such as H.W. Wilson publications Education Index and Socicd Science and 
Humanities Index. Not until chapter 10, in a discussion of Reference Guides to Special 
Subjects do the authors mention such specialized resources as Psychological Abstracts. 
Almost 15 years later, in 1983, in the fifth edition of the classic Guide to the Use of 
Libraries and Information Sources, Jean Key Gates continues the focus on the book by 
devoting the first eight chapters to discussing its role in library research. Chapter nine is 
devoted to indexes and here again the emphasis is on the Wilson indexes, but not 
exclusively the Readers ’ Guide. There is no mention of abstracting services, which are 
relegated to a section near the end of the book dealing with resources in special subject 
areas. By contrast, the 2005 edition of the Oxford Guide to Library Research, while it 
offers a couple of chapters on books, gets quite promptly and thoroughly into journal 
articles. It discusses the various vendors of indexing systems, has entire chapters on 
keyword searching and Boolean searching, and devotes a pair of chapters to citation and 
related record searching. 

In the discipline of education, where the ERIC system highlighted gray literature and 
began bringing journal article abstracts to its Current Index to Journals in Education in 
1969, journals and journal indexes were more prominent in the research guides, but 
books were still a clear focus. In 1975 Dorothea Berry wrote a short tome called A 
Bibliographic Guide to Educational Research. In it she directed researchers first to the 
library catalog — she spent 8 pages discussing how to fully utilize the information on the 
3x5 catalog cards. Next came a chapter on subject bibliographies, and then one on 
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indexes. That was followed by a discussion of abstracting services. In 1989, Lois Buttlar 
wrote Education: A Guide to Reference and Information Sources, and organized the 
book by subject area within the discipline. The suggestions started with bibliographies 
but indexes and abstracts to journal literature were the next group of resources covered. 

In 2003 Nancy O’Brien updated Buttlar’ s work. There are now hardly any bibliographies 
(literally only 2 titles) in the book. Indexes and Abstracts continue to play an important 
role and new to this edition is a section in most chapters on World Wide Web and 
Internet Sources. 

This progressively increasing importance of journal and online resources will not surprise 
many, but the extent to which books were central to research in the fairly recent past may 
be a surprise. The publishing record bears out the picture of transformation. 

Book publishing relative to journal publishing 

In the field of Education, books are decreasing in importance. WorldCat lists almost 
22,000 English language records for books about some aspect of education in 1995. In 
2005 there were about 13,000, not quite halved but a sharp decline in 10 years. Figures 
for other disciplines are included simply to show that the changes are not uniform across 
disciplines. 

Table 1: WorldCat Data on Book Catcdoging 





Education books 


Physics books 


Psychology Books 


1995 


21,937 


2,616 


9,868 


2005 


13, 212 


1,689 


10,608 


Difference 


-8,725 or 


-927 or 


+740 or 




-40% 


-35% 


+7% 



The rise in journal importance was not simply a result of decreases in book publishing. 
Throughout the period when new communication tools were increasing in popularity, 
journal publishing increased. Ulrich ’s International Periodicals Directory and its online 
equivalent UlrichsWeb.com provide data to investigate journal publishing levels in recent 
years. This series lists journals by subject. There are currently about 8,000 active 
journals in the education subjects. Just over 1,000, or about 16% are listed as being 
refereed. Different disciplines have different patterns of publication. As a comparison, 
Physics has about 2,000 active journals, nearly 700 or 37% refereed. The practitioner 
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interest in the field of education is reflected in the journal publication profile. A check of 
the Directory of Open Access Journals (DOAJ) for these two disciplines revealed that the 
number of journals listed there (177 in education and 39 in Physics) worked out to about 
2% of the total in both. It is difficult to draw comparisons across time using Ulrich’s; the 
online version is merely a current listing. Earlier printed volumes provide earlier figures 
to estimate the number of entries. There were about 4,000 education journals in the 
1989/1990 volumes of Ulrich’s and 6,700 in 2001. Clearly the trend is for an increasing 
number of journals being published. 



Table 2: Ulrich’s data on journal publishing 





Education 

Journals 


Refereed 

in 

Education 


Percent 

Refereed 


Physics 

Journals 


Refereed 
in Physics 


Percent 

Refereed 


Current 

data 


7,930 


1,298 


16% 


1,891 


697 


37% 


39*72001 


6,720 


828 


13% 


1404 


432 


31% 


28*71989- 

90 


4,050 





Why go to so much trouble to document what has become conventional wisdom? The 
numbers document an unexpected diversity between disciplines and a great deal of 
intradisciplinary change in relatively short timeframes. A reliance on journals changes 
the quality equation. When books were the primary source of information, 
recommendations in bibliographies and encyclopedia articles served as a means of 
explicitly designating quality. A student could feel confident using an item listed in such 
sources. Today these listings are viewed as being neither timely nor specific enough. 

Determinants of journal quality 

What is a quality journal? As journal literature has become a more important part of the 
literature review, peer review has been the most easily identified standard. How does an 
instructor communicate to students what to look for — how to identify high quality 
information? In an era when most journals are available online, telling them to look for a 
non-glossy publication without much advertising doesn’t work anymore. In the days 
when books were the basis of research, instructors relied on the disciplinary indexes, like 
Education Index or Socicd Sciences Index. These indexes cover only high quality-though 
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not necessarily only peer reviewed-publications. Their use was meant to supply a few 
journal resources to supplement the core reference list of books. Today many of the 
readily available indexes are a hodgepodge of everything from newsletters to top tier 
journals. Often these new products bring order to the chaos by labeling everything: this 
tab for newspapers, that one for peer-reviewed journals. Although imperfect the labels 
help students choose appropriate sources. As the demand for only peer reviewed items 
has grown; the discipline specific indexes-ERIC, PsycINFO, etc.-are also getting on the 
labeling bandwagon. Because journals were not such a central resource for earlier 
generations of researchers, being able to quickly identify peer review status was not as 
crucial. As indicated by the guides mentioned above, citations from the Readers Guide , 
which would have included popular magazines from Newsweek to the New Yorker , were 
considered acceptable articles to supplement books for undergraduates. Now even 
freshman writing projects often stipulate peer reviewed journals. 

As an aside, the problem of appropriate sources is compounded for online courses, which 
add access issues into the mix. The increased electronic subscription content can be both 
a blessing and a curse. On the plus side it makes it easier to assign a broad spectrum of 
readings to online students. But many faculty are reluctant to take on the role of 
technology guru as well as instructor. It is tempting load up courses with readings but 
avoid assigning actual library research projects to students, due to a fear of technical 
questions or access problems. Even when student investigation is required, instructors 
sometimes chose to sacrifice some aspect of quality to minimize access problems. I know 
of graduate level courses being instructed to find articles in JSTOR (thus sacrificing 
currency) or WilsonSelectPlus (sacrificing depth) for their “research.” These resources, 
while they supply ready links to the results generated, are not the quality indexes 
education students need to know. With so many sources of information at their disposal 
students require expert guidance in which to choose. Certainly at the graduate level, 
using indexes whose goal is to cover the field being researched should be a basic 
criterion. 

Labeling results or using only full text databases can be helpful for the freshman doing a 
paper, but does little for the grad student or new professional looking for a place to 
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submit a paper that is going to impress a promotion and tenure committee a few years 
later. How do scholars leam this type of information? 

A librarian’s personal vignette 

A few weeks ago I had a doctoral student ask me for a listing of the Tier I and Tier II 
journals in education. I explained that while we talk about such categories there is no 
formal listing. I offered her the O’Brien book mentioned above, which does a good job 
of listing top journals. I pointed out the Cabell listings, which try to give data like 
acceptance rate. I also pointed to several university web pages that make a stab at listing 
this for their tenure track faculty. We talked about the limitations, particularly for 
education researchers, of ISI Citation Indexes and their calculation of an Impact Factor. 
She was pretty indignant. How could universities require publication in top tier journals 
if that were not an identifiable thing? The woman has a point. We put a good deal of 
importance on publication record while asserting that quality is a matter of judgment. 
Some might argue it is professional consensus, but if so, it is an unspoken consensus. 
Reputation or prestige has become increasingly elusive not only because of the growing 
number of journals but also because the community of scholars has diversified around the 
globe. 

Beyond peer review: Citation counts 

Just as getting current publishing figures from WorldCat or UlrichsWeb.com was as easy 
as doing a quick search; the availability of online information makes quantifying the 
answer to questions of quality increasingly possible. Citation counts are one area that 
may become more important for education. For many years the best, nearly the only 
source of citation counts was the ISI product now called Web of Science. The Social 
Sciences Citation Index portion of that product covers education, but not well. In 2005 it 
covered 124 education journals. Compare that to the Science Citation Index's coverage 
of 297 physics journals. Remember there are currently 1,298 peer reviewed education 
journals being published and only 697 such journals in physics. 
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Table3: ISI Web of Science Citation Indexing Coverage 




Perhaps as a result of this poor coverage, citation counts and impact factors did not 
become as central to the definition of “quality” in the field of education as they have in 
some of the sciences. Partly this is because journal publication has not been the exclusive 
mode of publication but also this reflects the relatively poorer job that ISI has done in 
covering the field. Now citation indexing, as provided by Google Scholar, shows 
promise of covering the social sciences, including education, in greater depth than ISI has 
provided. Below are citation counts for nine articles from each of five different leading 
education journals. Three articles each from volumes for 1998, 2002 and 2004. While 
not a thorough comparison it does give a sense of the relative strength of the two 
resources. Google is doing a significantly better job. For most of the journals, ISI had 
slightly less than half of the citations Google found. Two caveats: first, citation counts 
are not impact factors 1 ; they indicate quality at the article not the journal level. Google is 
not yet aggregating data on journals. Second, Google is pulling in citations wherever it 
finds them, while ISI pulls them in only from printed, peer reviewed journals, but 
education as a field has never relied solely on journals. In an era of expanding 
publication platforms, the ISI reliance on refereed print journals becomes ever less 
defensible. 
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Table 4: ISI/Google Scholar comparison 



Journal 


ISI Citations 
3 articles in9 
issues 1998- 
2004 


Google Scholar 
citations same 
articles 


Difference 


ISI citations as 
a percentage of 
Google’s 


Journal of the 

Learning 

Sciences 


209 


611 


402 


34% 


Review of 

Educational 

Research 


238 


505 


267 


47% 


Learning and 
Instruction 


129 


281 


152 


46% 


American 

Educational 

Research 

Journal 


292 


695 


403 


42% 


Sociology of 
Education 


158 


225 


67 


70% 



Beyond peer review: Use statistics 

Another yardstick of quality that is becoming more important is use. Certainly citation 
counts have a similar basis but in an impatient, fast-moving era, why wait years for 
citations to appear when use counts are increasingly available and arguably valid, at least 
at the institutional level. As journals become an online resource available via institutional 
subscription they present another way to quantify quality. Once again the mechanical 
ability to count online activity provides the basic information, and aggregates journal 
importance in a profile customized to the institutional subscriber. The much-discussed 
tendency of citation counting to favor less specialized journals can be overcome by use 
statistics that will reflect the institution’s emphases. How can this be valid? Studies 
show that researchers are relying on institutional subscriptions to an ever-greater degree. 
Carol Tenopir and Donald King have been studying reading patterns in academe for 
thirty years. Their most recent study of faculty and students found that people are 
reading more and faster. Tenopir says: 

We first started noticing a strange phenomenon in the mid-1990s, where the 
growth rate of articles read and growth rate of total time spent reading that had 
pretty much followed each other proportionately, began to diverge. More 
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reading on average was occurring, but the total time spent reading was not 
increasing as much as would be expected. Reading now is much more likely to 
be skimming or scanning, as subject experts are reading more, but spending less 
time per reading. (Tenopir, 2006a, p.3) 

Where are readers getting this increased reading material? In another paper Tenopir 
notes: “The number of articles that they report reading has gone steadily up as e-journal 
collections grow. Over half of all article readings by faculty and up to three-quarters of 
readings by students are now from library e-collections. This willingness and need to read 
many articles is one measure of the value of e-collections” (Tenopir, 2006b). 

So far, use statistics for mainstream commercial journals are only reflective of use at a 
particular institution and are probably only seen by librarians unless there is a need to 
cancel. Born digital journals, which have had to fight for respect, are not always so 
reticent to tout their use. Many have counters or other similar indicators of the popularity 
of their articles posted on their web pages. 

Are use statistics a reliable measure of quality? 

Most established researchers view journal quality as a balance between editorial 
reputation, history of publishing highly useful articles, and perhaps the scholarly 
association publisher. Their assessments of particular titles develop over time. The 
suggestion that raw use numbers, especially those collected as online hits, indicate quality 
may boarder on the offensive. Newer scholars do not yet have a personal history with a 
variety of publications to draw upon. They are understandably reluctant to be completely 
reliant on the word of advisors. Unless one wishes to argue that scholars do not try to use 
high quality material, separating quality and use is difficult. Although the concept of 
high use as a basis of quality determinations is uncomfortable at the individual level, it is 
less problematic in the aggregate. For individual institutions, a particularly troublesome 
point may be the extent to which availability drives use. One recent article shows that at 
the institutional level use (as measured by library holdings statistics) correlates more 
closely with local citation counts than with the ISI impact factor. (Duy & Vaughan, 2006) 
When institutional holdings change significantly, local citation counts (what that 
institution’s faculty actually use and cite) change to reflect holdings. (Taylor 2007) 
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Technological change will put some traditional measures of quality on the wane. As more 
journals dispense with paper publication and become online only products, acceptance 
rates will decline in importance. Without the page limits of print, editors can publish 
every acceptable article. There will certainly be an incentive for them to do so, since the 
more activity a site attracts, the more likely its use, citation counts, and importance will 
rise. One could argue that high use, at least immediately after publication, can also mean 
the article is controversial, and this would need to be factored into use statistics. The 
same phenomenon would also impact citation counts. 

Publishers seem to be reacting to the increased importance of use levels. Scholars have 
documented how some journals in the sciences changed their article mix in ways meant 
to enhance their impact factor. (Cameron, 2005) Now some commercial publishers are 
changing their online presence in ways that will enhance their journals’ use levels. Many 
publishers signed with aggregators when ejoumals first became popular. ProQuest, 
EBSCO and to a lesser extent other vendors made their names by providing not just 
indexing - in fact one could argue that their indexing is generally inferior to that found in 
discipline specific databases - but the full text of the journals they were indexing. Now 
many commercial publishers are pulling their content out of the aggregator databases and 
providing unique interfaces to their journals. By providing an interface that searches only 
their journals they vie to keep users within their product line. Many work to pull in other 
content to make their sites attractive and keep users from going elsewhere. The recent 
agreement between Sage publications and the American Educational Research 
Association for Sage to publish the Association’s journals is an example. 

Global quality measures on the horizon 

The increasing importance of research voices from around the globe is a factor that will 
impact how quality is defined. The twentieth century publishing model tended to keep 
journal prices high and quality synonymous with exclusivity. The increasing trend 
toward open access will bring in new voices from other parts of the world. John 
Willinsky is one prominent voice on this front. Willinsky’s book The Access Principle 
(2005) in available online. His ideas about quality are intertwined with social justice 
issues rather compellingly. Willinsky believes firmly in value as indicated by use. 
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Citation counts can be one measure but he sees them as rather passe. If an item does not 
reach a large audience, how can it be valuable? Scholarship can only impact society if it 
is read and used. So he speaks out against high priced journals and restrictive copyright 
agreements. If we put it all up on the web, in open access journal articles or institutional 
repositories - it doesn’t much matter how we make things available - it will be found. 

Not just by researchers at the high prestige and well-funded institutions, but by 
researchers all over the world. 

I believe it is inevitable that use will be an increasingly important indicator of journal 
quality. Willinsky’s analysis of the economics of publishing have convinced me that our 
current system of high profit commercial publication needs reform and that open access 
is an important component of that reform. It remains to be seen whether the system of 
peer reviewed journals will adapt to meet changing needs or be superseded by a new 
model. 
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Endnote: 



1 Impact factor is defined as the average number of times articles from the journal 
published in the past two years have been cited in the given year. The impact factor is 
calculated by dividing the number of citations in a year by the total number of articles 
published in the two previous years. An impact factor of 1.0 means that, on average; the 
articles published one or two years ago have been cited one time. 

' Publish or Perish at http://www.harzing.eom/resources.htm#/pop.htm offers free 
software that will do personal or journal level citation analysis from Google Scholar. It is 
not yet particularly easy to use - even by citation analysis standards — but shows that such 
capabilities are imminent, not in the distant future 
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